Routing under heavy loading

ABSTRACT

According to the invention, a delivery network for assisting delivery of content objects over an Internet is disclosed. The delivery network includes a network outlet, an interface and a routing function. The network outlet is coupled to a plurality of full-route networks, where each of the plurality of full-route networks is capable of delivering content objects to a plurality of terminal networks. The plurality of terminal networks include a terminal network, where the plurality of terminal networks are coupled to a plurality of end user computers. The interface receives content objects for delivery to the plurality of end user computers. The routing function routes content objects in at least two modes, where a first mode routes content objects based upon a first route path from the network outlet to the terminal network, and a second mode routes at least some content objects using a second route path from the network outlet to the terminal network. The first route path is chosen based upon delivery efficiency. Switching from the first mode to the second mode is triggered when at least of a portion of the first route path reaches a predetermined level of use. The first and second route paths are different, and the second route path is less efficient than the first route path.

This application is a continuation of U.S. application Ser. No.13/102,941 filed on May 6, 2011, entitled “Routing Under Heavy Loading,”which is a continuation of U.S. application Ser. No. 11/461,173 filed onJul. 31, 2006, entitled Routing Under Heaving Loading,” which claims thebenefit of and is a non-provisional of U.S. Provisional Application No.60/761,582 filed on Jan. 23, 2006, entitled “Routing Under HeavyLoading,” and is a continuation-in-part of U.S. Pat. No. 7,706,280 filedon Aug. 1, 2005, entitled “Heavy Load Packet-Switched Routing,” all ofwhich are hereby incorporated by reference in their entirety for allpurposes.

BACKGROUND

This disclosure relates in general to content delivery and, morespecifically, but not by way of limitation, to content delivery underload.

Content originators provide content objects to recipients over theInternet. The Internet is an amalgamation of various networks who passeach other's network traffic such that recipients can receive theircontent objects. The various networks can be divided into Tier 1 andterminal networks. All Tier 1 networks are full-route networks such thatany point on the Internet can be reached by the Tier 1 network by usingterminal networks and other full-route networks. Each recipient hasInternet service from a terminal network, and content originators mayuse a combination of Tier 1 and terminal networks to deliver theircontent objects. The various networks, content originators andrecipients pay in some way for delivering content objects using Tier 1networks.

A content delivery network (CDN) is used by many content originators todeliver content more efficiently. The CDN may host, mirror or cache thecontent as well as deliver it to a requesting party. A web site ororigin server is linked to the CDN such that some or all content can besourced from the CDN rather than the content originator directly. CDNsalso use Tier 1 networks and may have peering relationships withterminal networks. This process of fulfilling a link through a CDN isusually transparent to the recipient.

Today, there are about 10-15 Tier 1 networks worldwide. Some can pass50-150 GB/min. to various destinations on the Internet. To accomplishthis, Tier 1 networks pass content objects to terminal networks and peerwith other Tier 1 networks. Peer Tier 1 networks agree to pass contentobjects from other Tier 1 networks destined to their terminal networkswithout cost. Tier 1 networks generally charge content providers andterminal networks for passing content objects such that peeringrelationships are generally avoided in these circumstances.

Peered Tier 1 networks agree to pass each other traffic only when theterminal network is not connected to the Tier 1 network receiving thetraffic initially, but the other peered Tier 1 network has a connectionto that terminal network. If a Tier 1 network receives a content objectdestined for a terminal network not directly connected to the Tier 1network, it is passed to a peer Tier 1 network with a connection to thatterminal network under the peering agreement.

Where a particular Tier 1 network has a connection to a terminalnetwork, a content object destined for that terminal network cannot bepassed to a peer Tier 1 network who may also have a connection to theterminal network. If the interconnect to the terminal network isoverwhelmed, the content object may be lost. Interconnects between Tier1 peers and terminal networks are expensive and tend to be over-built toaccommodate worst-case demand.

Singlecasting of large events can be difficult for CDNs or Tier 1networks to deliver effectively. Large events require content objects(e.g., files or streams) to tens of thousands of recipients in a shortperiod of time. Egress from the CDN and/or Tier 1 networks can beoverwhelmed by these large events. These egress points have finitebandwidth that serve as a bottleneck for large events. To avoid thesebottlenecks, CDNs and Tier 1 networks overbuild their egress points inanticipation of the loading. In some cases today, large events cannot beserved by any CDN or Tier 1 network without overloading parts of theirnetworks. For example, a news site reporting a major and unexpected newsevent can find that their delivery system cannot keep up with a suddenspike in demand.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is described in conjunction with the appendedfigures:

FIGS. 1A-1B are block diagrams of embodiments of a content system;

FIGS. 2A-2D are block diagrams of embodiments of the content system thatexposes routing details of the Internet;

FIG. 3 is a block diagram of an embodiment of Tier 0 network;

FIGS. 4A-4C are block diagrams of embodiments of a portion of thecontent system that shows interaction of networks in a particulargeographic region; and

FIGS. 5A-C are flow diagrams of embodiment of a process for deliveringcontent that switches between routing methods.

In the appended figures, similar components and/or features may have thesame reference label. Further, various components of the same type maybe distinguished by following the reference label by a dash and a secondlabel that distinguishes among the similar components. If only the firstreference label is used in the specification, the description isapplicable to any one of the similar components having the same firstreference label irrespective of the second reference label.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The ensuing description provides preferred exemplary embodiment(s) only,and is not intended to limit the scope, applicability or configurationof the invention. Rather, the ensuing description of the preferredexemplary embodiment(s) will provide those skilled in the art with anenabling description for implementing a preferred exemplary embodimentof the invention. It being understood that various changes may be madein the function and arrangement of elements without departing from thespirit and scope of the invention as set forth in the appended claims.

Specific details are given in the following description to provide athorough understanding of the embodiments. However, it will beunderstood by one of ordinary skill in the art that the embodiments maybe practiced without these specific details. For example, circuits maybe shown in block diagrams in order not to obscure the embodiments inunnecessary detail. In other instances, well-known circuits, processes,algorithms, structures, and techniques may be shown without unnecessarydetail in order to avoid obscuring the embodiments.

Also, it is noted that the embodiments may be described as a processwhich is depicted as a flowchart, a flow diagram, a data flow diagram, astructure diagram, or a block diagram. Although a flowchart may describethe operations as a sequential process, many of the operations can beperformed in parallel or concurrently. In addition, the order of theoperations may be re-arranged. A process is terminated when itsoperations are completed, but could have additional steps not includedin the figure. A process may correspond to a method, a function, aprocedure, a subroutine, a subprogram, etc. When a process correspondsto a function, its termination corresponds to a return of the functionto the calling function or the main function.

Moreover, as disclosed herein, the term “storage medium” may representone or more devices for storing data, including read only memory (ROM),random access memory (RAM), magnetic RAM, core memory, magnetic diskstorage mediums, optical storage mediums, flash memory devices and/orother machine readable mediums for storing information. The term“computer-readable medium” includes, but is not limited to portable orfixed storage devices, optical storage devices, wireless channels andvarious other mediums capable of storing, containing or carryinginstruction(s) and/or data.

Furthermore, embodiments may be implemented by hardware, software,firmware, middleware, microcode, hardware description languages, or anycombination thereof. When implemented in software, firmware, middlewareor microcode, the program code or code segments to perform the necessarytasks may be stored in a machine readable medium such as storage medium.A processor(s) may perform the necessary tasks. A code segment mayrepresent a procedure, a function, a subprogram, a program, a routine, asubroutine, a module, a software package, a class, or any combination ofinstructions, data structures, or program statements. A code segment maybe coupled to another code segment or a hardware circuit by passingand/or receiving information, data, arguments, parameters, or memorycontents. Information, arguments, parameters, data, etc. may be passed,forwarded, or transmitted via any suitable means including memorysharing, message passing, token passing, network transmission, etc.

With reference to FIG. 1A, an embodiment of a content system 100-1 isshown. The content originator 106 produces content objects. Included inthe content originator 106 are a content provider 108 and a contentorigin site or web site 116. A content object is any content file orcontent stream and could include, for example, software, audio, video,pictures, data, and/or text. The content object could be live, delayedor stored. The content site 116 can be located within the infrastructureof the content provider 108 and/or at an alternative location.Throughout the specification, reference may be made to a content object,content stream and/or content file, but it is to be understood thatthose terms could be used interchangeably wherever they may appear.

The content originator 106 is the source or re-distributor of contentobjects. The content site 116 is an Internet site accessible directly orindirectly via the Internet 104 by the recipient computer 128. Contentobjects from the content provider 108 are made available to recipients112 through the content site 116. In one embodiment, the content site116 could be a web site where the content is viewable with a webbrowser. In other embodiments, the content site 116 could be accessiblewith application software other than a web browser and/or accessiblefrom devices other than personal computers. In some cases, recipientcomputers 128 can act as content originators 106 and content originators106 can act as recipient computers 128.

The recipient computer 128 receives the content object and processes itfor the recipient 112. The recipient computer 128 could be a personalcomputer, media player, handheld computer, Internet appliance set topbox, phone, or any other device that can receive content objects. Insome cases, the recipient computer 128 can be a number of computingdevices that may be networked together.

Each recipient computer or other device 128 is associated with anInternet service provider (ISP) or terminal network. The ISP is part ofthe Internet 104 in FIG. 1A. Each ISP provides Internet connectivity toone or more recipient computers or other devices 128. A recipientcomputer or other device 128 requests and accepts the content objectsfor realization to the recipient 112. The ISP works with other networksof the Internet 104 to deliver content between the content originator106 and the recipient computer 128.

With reference to FIG. 1B, an embodiment of a content system 100 isshown where a content originator 106 offloads the delivery of thecontent objects to a content delivery network (CDN) 110. The CDN 110 isused to offload some or all content object deliveries from the contentoriginator 106. Embodiments can use multiple CDNs 110 or could have aCDN 110 integral to the content originator 106. The content site 116 canbe located within the infrastructure of the content provider 108, withina CDN 110 and/or at an alternative location.

Many content providers 108 use a CDN 110 to deliver the content objectsto customers or recipients. When a content object is requested by arecipient, the CDN 110 retrieves the content object from the contentprovider 108. Alternatively, the content provider 108 may directlyprovide the content object to the CDN 110, i.e., in advance of the firstrequest. The CDN 110 then provides the content object to the recipient112. The content provider 108 typically pays the CDN 110 for thedelivery of the content object. In other embodiments, the CDN 110 couldbe captive or associated with the content provider 108 such that paymentis not performed. Links on the content site 116 and/or links toindividual content objects are structured to allow delivery through oneor more CDNs 110. The links may be rewritten before a web page isrendered or after a link is activated by using a redirect, for example.

With reference to FIG. 2A, a block diagram of an embodiment of a contentsystem 200-1 is shown that exposes routing details of the Internet 104.This embodiment shows the complex relationships between various networks212, 222, 224 that make up the Internet 104. Many other interconnectionconfigurations are possible and this embodiment is simplified in thatthe many of connections and networks are not depicted for clarity.

Recipient computers 128 get their Internet access through a terminalnetwork 224. The last network involved in the delivery to the recipientcomputer 128 is the terminal network 224. Terminal networks 224 arecommonly called Internet service providers (ISPs). Generally, contentoriginators 106 pass content objects to the recipient computers 128using the Internet 104, which ultimately includes a terminal network224.

The Internet 104 is largely a group of networks 212, 222, 224 that agreeto carry each-others network traffic for free (e.g., a peerrelationship) or some fee. These networks 212, 222, 224 include Tier 1or full-route networks 222, Tier 0 networks 212 that use multiplefull-route networks and terminal networks 224. Interconnects between thevarious networks 212, 222, 224 could be implemented with packet switchedor circuit switched connections that are generally bandwidth controlledor limited. For example, the first content originator 106-1 could have a10 MB/min. connection a first Tier 1 network 222-1. Over utilizedinterconnects generally suffer poor QoS that could include slow speedsand/or lost packets.

Tier 1 networks 222 are networks that typically charge to receive orsend content objects from CDNs 110, terminal networks 224 and contentoriginators 106. Tier 1 networks 222 are full-route networks in that anyInternet protocol (IP) address is reachable from any Tier 1 network 222.Terminal networks 224 use the full-route ability of the Tier 1 networks222 to provide full-route service to the recipient computers 128 whosubscribe with the terminal network 224 for Internet service.Conversely, content originators 106 and CDNs use the full-route serviceto deliver content to any IP address.

The CDN 110 and content originators 106 may have different arrangementswith the various networks 212, 222, 224 that make up the Internet 104.Because of these differing arrangements, the delivering costs to eachCDN 110 could vary for a particular recipient 112 or route chosen to therecipient 112. In this embodiment, the CDN 110 has two egress points,namely, a first egress point goes to a first Tier 0 network 212 and asecond egress point goes to a third Tier 1 network 222-3. Because of theCDN's 110 relationship with the Tier 0 network 212 and the third Tier 1network 222-3, either can be used by the CDN 110 to deliver contentobjects. The first content originator 106-1 has arrangements with afirst and second Tier 1 networks 222-1, 222-2 for communication with theInternet and delivery of content objects. The second content originator106-2 uses the Tier 0 network 212 for communication and delivery.

Each Tier 1 network 222 can route to all the terminal networks 224, butthe efficiency to a particular end point would vary for each Tier 1network 222. All Tier 1 networks generally have peering relationshipswith other Tier 1 networks, but a first Tier 1 network 222-1 cannot passtraffic to another Tier 1 network 222-2 where the first Tier 1 network222-1 has egress to the terminal network 224 associated with the targetrecipient computer 128 for the traffic. For example, when the secondTier 1 network 222-2 receives a content object for delivery to the thirdterminal network 224-3, the content object can be passed to the first orthird Tier 1 networks 222-1, 222-3 for delivery because the second Tier1 network 222-2 has no direct egress to the third terminal network224-3. In another example, the first Tier 1 network 222-1 may receive acontent object for delivery to the second terminal network 224-2, buthas to try to deliver the content object to the second terminal network224-2 even if the interconnect to between the two is loaded so as toprovide inadequate QoS.

The Tier 0 network 212 generally has interconnects with at least twoTier 1 networks 222. In this embodiment, the Tier 0 network 212 hasinterconnects with three Tier 1 networks 222. The Tier 0 network 212could also have interconnects to various terminal networks 224 in otherembodiments. Both content originators 106 and CDNs 110 may use the Tier0 network 212 for delivery instead of a Tier 1 network 222 or terminalnetwork 224. In some cases, a Tier 1 network 222 could even use the Tier0 network 212 for delivery. For example, the second Tier 1 network 222-2may have a content object for the second terminal network 224-2, but thesecond Tier 1 network 222-2 determines that its interconnect to thesecond terminal network 224-2 would provide inadequate QoS. The secondTier 1 network 222-2 could pass the content object to the Tier 0 network212 for delivery by the Tier 0 network 212 using the first or thirdterminal networks 224-1, 224-3.

The Tier 0 network 212 can route in at least two modes. In a first mode,the Tier 0 network 212 routes with the three Tier 1 networks usingefficiency to guide the routing. Efficient routing is typically basedupon the number of hops and the speed of the routers between those hops.For example, the efficient or optimal paths may be chosen based uponlatency or other conventional techniques. The first mode corresponds toconventional routing methods.

At some point, one of the efficient interconnects may become utilizedbeyond a threshold, which can trigger routing in the second mode. TheTier 0 network 212 can communicate traffic using any of the threeconnected Tier 1 networks 222. If one interconnect gets loaded beyond athreshold, the other interconnects can be used even if not optimal.Since any of the Tier 1 networks 222 is full route, giving a contentobject to a sub-optimal Tier 1 network 222 for delivery will stillresult in delivery to the recipient computer 128 even if at a lower QoS.A sub-optimal route is defined as one that is less efficient under lightload conditions and would not be chosen by conventional algorithms thatbase decisions on efficiency.

Switching between routing modes is demonstrated in the followingexample. The interconnect between the Tier 0 network 212 and the thirdTier 1 network 222-3 may be capable of 50 MB/min., for example. The mostefficient path to recipient computers 128-3 of the third terminalnetwork 224-3 uses the third Tier 1 network 222-3 in this embodiment.When the interconnect between the Tier 0 network 212 and the third Tier1 network 222-3 reaches 40 MB/min. utilization threshold, for example,the second routing method could be initiated. The Tier 0 network 212 inthe second routing mode could start routing to recipient computers 128-3of the third terminal network 224-3 using the first and second Tier 1networks 222-1, 222-2 even though they are less efficient routes to thethird terminal network 224-3 under normal circumstances.

Routing in the second mode can be done in a number of ways. In oneembodiment, all routing is done in a random or mixed-up manner. When anyone interconnect reaches a threshold, all future content objects aredelivered to Tier 1 networks 222 chosen in a manner that doesn't takeinto account routing optimization as was done in the first mode. Theinterconnects could be randomly used, assigned in a sequential fashion,assigned to the lowest utilized interconnect or some other non-optimalrouting method. Generally, the Tier 0 network 212 switches tosub-optimal routing under heavy loads where that sub-optimal routingassures that any bottleneck to the optimal Tier 1 networks 222 isavoided as long as possible. Once utilization falls below a secondthreshold, the Tier 0 network 212 can resume routing with the firstrouting mode once again.

Referring next to FIG. 2B, a block diagram of another embodiment of acontent system 200-2 is shown that exposes routing details of theInternet 104. In this embodiment, the Tier 0 network 212 hasinterconnections to two Tier 1 networks 222-1, 222-2 and one terminalnetwork 224-3. The terminal network 224-3 receives connectivity from thefirst Tier 1 network 222-1, the second Tier 1 network 222-2 and the Tier0 network 212. The third terminal network 224-3 may pay the Tier 0network 212 for connectivity. When the Tier 0 network 212 is routing tothe third terminal network 224-3, routing in the first mode would use adirect interconnect between the Tier 0 network 212 and the terminalnetwork 224-3. Switching to the second mode of routing would use thefirst and second Tier 1 networks 222-1, 222-2 as additional routes tothe third group of recipient computers 128-3.

A Tier 1 network 222 may arrange with the Tier 0 network 212 to delivercontent in one embodiment. For example, where the first Tier 1 network222-1 receives a content object for delivery through the first terminalnetwork 224, the first Tier 1 network 222-1 may be determine itsinterconnect to the first Tier 1 network 222-1 is over-utilized. Thefirst Tier 1 network 222-1 could hand-off the content object delivery tothe Tier 0 network 212 who could pass it to the second Tier 1 network222-2, for example, for delivery. The Tier 0 network 212 may charge forthis ability or could net the charge against the charges for using thefirst Tier 1 network 222-1 to deliver other content objects.

Referring next to FIG. 2C, a block diagram of yet another embodiment ofa content system 200-3 is shown that exposes routing details of theInternet 104. In this embodiment, there is no Tier 0 network, but Tier 1plus networks 226 allow routing in the second mode with other Tier 1plus networks 226. Tier 1 networks 222 do not route traffic destined forterminal networks that they have an interconnection to, but a Tier 1plus network 226 would do that routing if the connection between theTier 1 plus network 226 and the terminal network 224 were loaded beyonda threshold.

The differences between a Tier 1 plus network 226 and a Tier 1 network222 can be demonstrated by an example. In this example, the CDN 110receives a content object destined for the third group of recipientcomputers 128-3 by way of the third terminal network. Should the CDN 110choose to send the content object using the Tier 1 network 222, anybottleneck between the Tier 1 network 222 and the terminal network 224-3could affect QoS. Alternatively, the CDN 110 could use the second Tier 1plus network 226-2. Should the connection between the second Tier 1 plusnetwork 226-2 and the third terminal network 224-3 come utilized beyondsome threshold and/or QoS begin to suffer, the second Tier 1 plusnetwork 226-2 could pass the content object to the first Tier 1 plusnetwork 226-1 to utilize the connection between the first Tier 1 plusnetwork 226-1 and third terminal network 224-3.

Referring next to FIG. 2D, a block diagram of still another embodimentof a content system 200-4 is shown that exposes routing details of theInternet 104. In this embodiment, there are no Tier 1 networks depictedsuch that all are Tier 1 plus networks 226. Any CDN 110 or contentoriginator 106 need only have a connection with a single Tier 1 plusnetwork 226 in this embodiment. So long as the connection to the Tier 1plus network 226 isn't overwhelmed, there is likely to be adequate QoSfor other parts of the Internet 104.

With reference to FIG. 3, a block diagram of an embodiment of a Tier 0network 212 is shown. This embodiment has a central POP trafficdistributor 316 and content request interface 314 coupled by a WAN 320or other backbone to a number of remote POPs 304. Various embodimentscould have any number of POPs 304 geographically distributed to delivercontent. Each POP 304 has connections with other networks 222, 224, 226to communicate with recipient computers 128.

There can be multiple interconnects between a Tier 0 network 212 andother networks 222, 224, 226. Multiple connections may be dispersedamong the multiple POPs 304. For example, the Tier 0 network 212 mayconnect to a Tier 1 network 222 in POPs 304 in Chicago and Miami, butnot in a POP 304 in Los Angeles. Where there are multipleinterconnections, the Tier 0 network can treat them separately orcollectively when operating in the first and second routing modes. Forexample, when the Chicago interconnect with a Tier 1 network 222 becomesutilized beyond a threshold, the Miami connection to the same Tier 1network 222 could be used in mode two routing along with connections toother Tier 1 networks 222.

Routing decisions are made in a POP traffic distributor 316 and/or inthe POP 304. In one embodiment, the POP traffic distributor 316 choosesthe POP 304 and the POP 304 chooses the egress point. In anotherembodiment, the POP traffic distributor 316 makes all the routingdecisions within the CDN 110. Other embodiments could directly have thePOPs 304 decide independently if they will deliver the content object orpass the delivery off to another POP 304.

The POP traffic distributor 316 receives requests for content objectsthrough a content request interface 314 and distributes those requeststo a POP 304 best suited to service the request. The “best suited” POPmay be different when routing in mode one or mode two. Communicationbetween the POP traffic distributor 316 and the POPs takes place over aWAN backbone 320 (e.g., leased line, a private network and/or theInternet 104). In alternative embodiments, the WAN backbone 320 could bereplaced with a tunneled connection over the Internet 104 or atraditional Internet connection.

In this embodiment, there are three POPs 304 that serve requests forcontent. The POPs 304 each have connections to various Tier 1 networks222, 226 and terminal networks 224 to serve various recipient computers128. Different POPs 304 could communicate with different networks. ThePOP traffic distributor 316 is aware of the communication options foreach POP 304 along with the terminal network 220, 224 associated with aparticular recipient computer 128. So long as a particular POP 304and/or connection is not utilized beyond a threshold, the POP trafficdistributor 316 will include that POP 304 as a possible choice fordelivering a particular content object. Once the content object requestis associated with a particular POP 304, it is served or streamed fromthat POP 304. That is to say that a particular content object is notdivided among multiple POPs 304 for delivery in this embodiment.

Referring next to FIG. 4A, a block diagram of an embodiment of a portion400-1 of the content system 100 is shown that shows interaction ofnetworks in a particular geographic region. This embodiment could haveadditional geographic regions with additional and redundant connectionsbetween networks, but the simplified embodiment shown in this figureonly illustrates networks in a particular geographic region. Multiplecontent originators 106 are coupled to a Tier 0 network 212. In one POP,the Tier 0 network 212 is interconnected with three Tier 1 networks 222and a terminal network 224-4. The connection between from Tier 0 network212 to the terminal network 224 may be preferred over use of any of theTier 1 networks 222 as the fourth terminal network 224 pays fordelivered content in this embodiment.

The various routing paths are shown in Table I for this embodiment. Thehops correspond to a routing operation between two interconnects for usein routing between the content originator 106 and a recipient group.Each extra hop generally increases latency and increases the risk ofreaching a bottleneck. In routing to the first recipient group 128-1,for example, one route has three hops and the other routes have fourhops. Routing in the first mode may be based upon minimizing hops, butshould the connection between the Tier 0 network 212 and the first Tier1 network 222-1 become utilized beyond a threshold routing could switchto a second mode. In the second mode, all three routes could be used inround-robin or random fashion such that all future delivery streams aredivided among all possible paths. Alternatively, the second mode couldremove the over-utilized route(s) and use the remaining routes untilutilization decreases for the excluded routes.

TABLE I Routing Possibilities to the Groups of Recipient ComputersRecipient Group Various Routes 128-1 106-x, 212, 222-1, 224-1, 128-1106-x, 212, 222-2, 222-1, 224-1, 128-1 106-x, 212, 222-3, 222-1, 224-1,128-1 128-2 106-x, 212, 224-2, 128-2 106-x, 212, 222-1, 224-2, 128-2106-x, 212, 222-2, 222-1, 224-2, 128-2 106-x, 212, 222-3, 222-1, 224-2,128-2 128-3 106-x, 212, 222-1, 224-3, 128-3 106-x, 212, 222-2, 224-3,128-3 106-x, 212, 222-3, 224-3, 128-3 128-4 106-x, 212, 224-4, 128-4106-x, 212, 222-2, 224-4, 128-4 106-x, 212, 222-1, 222-2, 224-4, 128-4106-x, 212, 222-3, 222-2, 224-4, 128-4 128-5 106-x, 212, 222-3, 224-5,128-5 106-x, 212, 222-1, 222-3, 224-5, 128-5 106-x, 212, 222-2, 222-3,224-5, 128-5 128-6 106-x, 212, 222-2, 224-6, 128-6 106-x, 212, 222-3,224-6, 128-6 106-x, 212, 222-1, 222-2, 224-6, 128-6 106-x, 212, 222-1,222-3, 224-6, 128-6

With reference to FIG. 4B, a block diagram of an embodiment of a portion400-2 of the content system 100 is shown that shows interaction ofnetworks in a particular geographic region. This embodiment has two Tier1 plus networks 226 and one Tier 1 network 222 in this geographic regionused by the terminal networks 224-1 and content originator 106. Table IIshows the various routing options available. The interlink between thetwo Tier 1 plus networks can be used even where the Tier 1 plus networkhas a direct connection with the target terminal network 224. Table IIshows in italics where Tier 1 plus networks 226 are taking advantage ofthe interlink in the second mode of routing.

Tier 1 plus networks 226 may account for traffic consumed in the secondmode of routing. This traffic could be charged for in a manner beyondthe peering agreement. For example, the first Tier 1 plus network 226-1could be charged for passing traffic to the second Tier 1 plus network226-2 that is destined for the third terminal network 224-3. Conversely,the second Tier 1 plus network 226-2 could be charged for passingtraffic to the first Tier 1 plus network 226-1 that is destined for thethird terminal network 224-3. In one embodiment, the various chargescould be netted together to partially cancel some before payments aremade periodically.

In this embodiment, the Tier 1 plus or Tier 1 networks 222, 226 onlypass traffic to another Tier 1 plus or Tier 1 network 222, 226 when thereceiving network 222, 226 has a connection to the target network 224.Other embodiments could allow passing traffic regardless of whetherthere is a connection. For example, the following route between thefirst content originator 106-1 and the fifth group of recipientcomputers 128-5 is possible: 106-1, 226-1, 226-2, 222, 224-5, 128-5.Movement of traffic in this way could only be done when the secondrouting mode is triggered for one embodiment.

TABLE II Routing Possibilities to the Groups of Recipient ComputersRecipient Group Various Routes 128-1 106-1, 226-1, 224-1, 128-1 106-2,226-2, 226-1, 224-1, 128-1 106-3, 226-2, 226-1, 224-1, 128-1 106-4, 222,226-1, 224-1, 128-1 106-4, 226-1, 224-1, 128-1 128-2 106-1, 226-1,224-2, 128-2 106-2, 226-2, 226-1, 224-2, 128-2 106-3, 226-2, 226-1,224-2, 128-2 106-4, 226-1, 224-2, 128-2 106-4, 222, 226-1, 224-2, 128-2128-3 106-1, 226-1, 224-3, 128-3 106-1, 226-1, 226-2, 224-3, 128-3106-2, 226-2, 224-3, 128-3 106-2, 226-2, 226-1, 224-3, 128-3 106-3,226-2, 224-3, 128-3 106-3, 226-2, 226-1, 224-3, 128-3 106-4, 226-1,224-3, 128-3 106-4, 226-1, 226-2, 224-3, 128-3 106-4, 222, 224-3, 128-3128-4 106-1, 226-1, 226-2, 224-4, 128-4 106-2, 226-2, 224-4, 128-4106-3, 226-2, 224-4, 128-4 106-4, 226-1, 226-2, 224-4, 128-4 106-4, 222,226-2, 224-4, 128-4 128-5 106-1, 226-1, 222, 224-5, 128-5 106-2, 226-2,222, 224-5, 128-5 106-3, 226-2, 222, 224-5, 128-5 106-4, 226-1, 222,224-5, 128-5 106-4, 222, 224-5, 128-5 128-6 106-1, 226-1, 226-2, 224-6,128-6 106-1, 226-1, 222, 224-6, 128-6 106-2, 226-2, 224-6, 128-6 106-3,226-2, 224-6, 128-6 106-4, 226-1, 226-2, 224-6, 128-6 106-4, 226-1, 222,224-6, 128-6 106-4, 222, 224-6, 128-6

Referring next to FIG. 4C, a block diagram of an embodiment of a portion400-3 of the content system 100 is shown that shows interaction ofnetworks in a particular geographic region. In this embodiment, thereare three Tier 1 plus networks 226 used for delivering content in thisgeographic region. Table III shows routing available for the variousgroups of recipient computers 128. The routes shown in italics are madeavailable by using Tier 1 plus networks 226 instead of convention Tier 1networks 222.

TABLE III Routing Possibilities to the Groups of Recipient ComputersRecipient Group Various Routes 128-1 106-1, 226-1, 224-1, 128-1 106-2,226-2, 226-1, 224-1, 128-1 106-3, 226-2, 226-1, 224-1, 128-1 106-4,226-3, 226-1, 224-1, 128-1 128-2 106-1, 226-1, 224-2, 128-2 106-2,226-2, 226-1, 224-2, 128-2 106-3, 226-2, 226-1, 224-2, 128-2 106-4,226-3, 226-1, 224-2, 128-2 128-3 106-1, 226-1, 224-3, 128-3 106-1,226-1, 226-2, 224-3, 128-3 106-1, 226-1, 226-3, 224-3, 128-3 106-2,226-2, 224-3, 128-3 106-2, 226-2, 226-1, 224-3, 128-3 106-2, 226-2,226-3, 224-3, 128-3 106-3, 226-2, 224-3, 128-3 106-3, 226-2, 226-1,224-3, 128-3 106-3, 226-2, 226-3, 224-3, 128-3 106-4, 226-3, 224-3,128-3 106-4, 226-3, 226-2, 224-3, 128-3 106-4, 226-3, 226-3, 224-3,128-3 128-4 106-1, 226-1, 226-2, 224-4, 128-4 106-2, 226-2, 224-4, 128-4106-3, 226-2, 224-4, 128-4 106-4, 226-3, 226-2, 224-4, 128-4 128-5106-1, 226-1, 226-3, 224-5, 128-5 106-2, 226-2, 226-3, 224-5, 128-5106-3, 226-2, 226-3, 224-5, 128-5 106-4, 226-3, 224-5, 128-5 128-6106-1, 226-1, 226-2, 224-6, 128-6 106-1, 226-1, 226-3, 224-6, 128-6106-2, 226-2, 224-6, 128-6 106-2, 226-2, 226-3, 224-6, 128-6 106-3,226-2, 224-6, 128-6 106-3, 226-2, 226-3, 224-6, 128-6 106-4, 226-3,224-6, 128-6 106-4, 226-3, 226-2, 224-6, 128-6

With reference to FIG. 5A, a flow diagram of an embodiment of a process500-1 for delivering content is shown that switches between routingmethods. The depicted portion of the process begins in step 504 where acontent object request is received by the Tier 0 network 212 or Tier 1plus network 226. This could be in the form of a uniform resourceidentifier (URI) or uniform resource locator (URL) that indicates thecontent object desired. The Tier 0 network 212 or Tier 1 plus network226 can determine an IP address of the requesting computer 128 from theprotocol level handshake to pass the URI. The IP address corresponds toa terminal network 224 and a general geographic region. From thisinformation, peering relationships, POP geographical locations andinterconnect points, are analyzed to determine the possible POPs 304 touse.

In step 508, any POPs 304 and interconnect points utilized overthreshold can be removed from consideration. A saturated POP may becompletely saturated at all interconnect points or partially saturatedat the interconnect to the relevant peering network 222, 226 or terminalnetwork 224. For example, routing to the sixth group of recipientcomputers 128-6 resulting in possible routes given in Table I below fortwo geographically diverse POPs 304. In this example, the recipientcomputer 128-6 is located in Phoenix, Ariz. The first POP 304-1 is nearTucson, Ariz. and corresponds to FIG. 4A, and the second POP 304-2 isnear New York, N.Y. and corresponds to FIG. 4C. The sixth terminalnetwork 224-6 can pass traffic between different geographic regions suchthat traffic received in New York, N.Y. could be passed to a recipientcomputer 128-6 in Tucson, Ariz. or elsewhere.

In this embodiment, the threshold for over-utilization is set at 80%.Since the all routes of the first POP 304-1 to the recipient computer128-6 are saturated beyond the threshold, the first POP 304-1 would beremoved from consideration although geographically closer to therecipient computer 128-6. This embodiment only considers utilization ofbandwidth, but other embodiments consider utilization of other resourcesthat could affect QoS. For example, if 90% of processing resources areactive, the POP could be removed from consideration.

TABLE IV Routing Possibilities to Fourth Group of Recipient ComputersPOP(s) Possible Routes Saturation 304-1 in 106-x, 212, 222-2, 224-6,128-6 90% Tucson, AZ 106-x, 212, 222-3, 224-6, 128-6 85% 106-x, 212,222-1, 222-2, 224-6, 128-6 92% 106-x, 212, 222-1, 222-3, 224-6, 128-687% 304-2 in 106-1, 226-1, 226-2, 224-6, 128-6 95% New York, NY 106-1,226-1, 226-3, 224-6, 128-6 35% 106-2, 226-2, 224-6, 128-6 91% 106-2,226-2, 226-3, 224-6, 128-6 92% 106-3, 226-2, 224-6, 128-6 94% 106-3,226-2, 226-3, 224-6, 128-6 33% 106-4, 226-3, 224-6, 128-6 30% 106-4,226-3, 226-2, 224-6, 128-6 94%

In step 512, the most efficient egress point is determined. In thisexample, the second POP 304-2 is the only remaining POP 304 underconsideration, but in other embodiments many more POPs could beavailable such that interconnects from multiple POPs are considered. Inthis example, the third content originator 106-3 is the source of thecontent object being delivered to the sixth recipient computer 128-6.There are two routing possibilities between the third content originator106-3 and the second POP 304-2, but one has two hops and the other hasthree. In this embodiment, the more efficient routes are those with thefewest hops, but other embodiments could determine efficiency in otherways. The request for the content object is passed to the second POP304-2 in step 516.

Some embodiments have lower cost routes such as terminal networks 224.For example, the Tier 0 network 212 in FIG. 2B can route to both Tier 1networks 222 or terminal networks 224-3, but the terminal network 224-3is likely to be lower or no cost. This embodiment tries to route to thelower cost networks in steps 518 and 520, before using full-routenetworks (e.g., Tier 1 and Tier 1 plus networks 222, 226) in step 528,which are typically more expensive. The decision in step 518 passes therequest to a terminal network 224 in step 520 where there is a peeringrelationship or lower-cost network, for example, or to a full-routenetwork in step 528 if there is no terminal network 224. Where there isno network with a cost advantage, the request is passed from step 518 tostep 528 to choose the most efficient full-route network for delivery.In this embodiment, once delivery starts with a particular route,delivery continues on that route until the stream or file delivery iscompleted.

For step 520, a saturation threshold is set at some number. Initially,saturation is determined by figuring how much of the bandwidth of theinterconnect is consumed in one embodiment. If packet loss increasesbeyond some level, saturation would be found even if the egress pointbandwidth is not completely consumed as packet loss is an indicator thatthere is another bandwidth bottle neck between the egress point and thedestination computer 128. Where the terminal network 224 is determinedto be saturated, processing goes to step 528 to find a full-routenetwork route instead of step 534 to deliver the content object with theterminal network 224.

Where the peer network connection is saturated as determined in step520, an alternative full-route Tier 1 or Tier 1 plus network 222, 226would be found in step 528. In this example, the 106-3 to 226-2 to 224-6to 128-6 is found to be the most efficient as it has the least hops. Theaffected POP 304-1 would switch to delivering all new traffic for thesixth group of recipient computers 128-6 to this path until thesaturation level of the first POP 304-1 decreases.

The chosen route is checked for saturation in step 524 by checking forover-utilization from the second Tier 1 plus network 226-2. Otherembodiments could check the whole path to find bottlenecks anywhere inthe route. The saturation determination is a function of boththeoretical bandwidth of the egress point and/or packet loss. Excessiveuse of the bandwidth or observed packet loss would result in adetermination in step 524 that the interconnection between the Tier 1plus network 226-2 and the sixth terminal network 224-6 isover-utilized. In this embodiment, the utilization threshold is 90% andthe 106-3 to 226-2 to 224-6 to 128-6 route is utilized to 94%. Werethere utilization below the threshold, processing would pass from step524 to step 534 for delivery of the content object.

Where the interconnection between the Tier 1 plus network 226-2 and thesixth terminal network 224-6 is over-utilized as shown in the example ofTable IV, the content object would be delivered by some other route ifavailable. In step 536, a list of alternative routes is determined androuting switches to the second mode in step 540. In this example, theroute of 106-3 to 226-2 to 226-3 to 224-6 to 128-6 is an alternativethat uses three hops instead of the more efficient two hop andover-utilized route.

The traffic that would be routed to the over-utilized route is divertedto the less efficient, but less utilized route in the second routingmode in step 544. Where there are a number of alternatives, contentobjects can be divided between them in random or mixed-up fashion so asto generally distribute the loading. There are many different possiblealgorithms to choose other routes in step 544. These algorithmsgenerally distribute traffic across various networks 222, 224, 226 inone or more POPs 304 so long as other routes are not also saturated. Forexample, a first overflow request could be served by a first POP 304 andthe next overflow request could be served by a second POP 304.

Some embodiments only route overflow to other Tier 1 or Tier 1 plusnetworks 222, 226 associated with the POP 304 where the saturationoccurred instead of considering other POPs 304. Some algorithms couldweight the attractiveness of a route according to utilization, costand/or efficiency. None of these algorithms is based solely uponefficiency as the most efficient network 222, 224, 226 is alreadysaturated. A first routing mode takes all the unsaturated routes anddetermines the most efficient one or more. After over-utilized, thefuture content object requests are distributed randomly or sequentiallyacross all routes of the same cost. Where there are several cost levels,one algorithm weights the attractiveness of a route according to therelative cost. Another algorithm distributes requests according tosaturation level such that the least saturated are favored over the moresaturated. Yet another algorithm takes all routes and distributestraffic among them. Once the alternative route is chosen in step 544,the request is fulfilled in step 534.

With reference to FIG. 5B, a flow diagram of another embodiment of aprocess 500-2 for delivering content is shown that switches betweenrouting methods. This embodiment eliminates the link between steps 520and 528 and does not find an efficient full-route network 222, 226 afterfinding the terminal network 224 is saturated. Processing goes from step520 to step 536 when the terminal network 224 is saturated. Accordingly,where the first chosen terminal network 224 or full-route network 222,226 is saturated, the alternatives are analyzed to find otherpossibilities. In one embodiment, this has the effect of distributingthe traffic across many of the alternative paths.

With reference to FIG. 5C, a flow diagram of yet another embodiment of aprocess 500-3 for delivering content is shown that switches betweenrouting methods. This embodiment does not differ from the embodiment ofFIG. 5A until after step 516. In step 530, the most efficient route isdetermined, which could be egress to a terminal network 224 a Tier 1network 222 or a Tier 1 plus network 226. If the first choice is notsaturated in step 522, the content object is delivered in step 534.

When the initial route is saturated, processing continues to step 538where alternative routes are determined. The alternatives may be chosenfrom the present POP 304, all possible POPs 304 and/or all unsaturatedPOPs 304. In step 542, the affected POP switches to routing based uponfactors other than efficiency. For example, the routing could besequential or randomly disbursed among the alternatives, where thealternatives are weighted by cost, saturation level and/or efficiency.The alternative for a particular request is chosen in step 546 anddelivered in step 534. In one embodiment, switching to the alternativerouting would distribute excess to other routes that could deliver apiece of content.

Although the embodiments are described above in terms of saturation,switching to the alternative routing method or second mode could be donefar before saturation. For example, switching could be any thresholdsuch as 40%, 50%, 60%, 70%, or 80% utilization. Use of the termsaturation is not necessarily meant to imply that performance isdegraded. Indeed, performance might not be affected until 95% or moresaturation in some embodiments. Switching to alternative routing at 50%utilization would serve to avoid any premature risk of degradedperformance due to saturation. In any case, the saturation threshold canbe set to any value in various embodiments, for example, 40%, 50%, 60%,70%, 80%, 90%, 95%, or 100%.

Some of the above embodiments talk of the Tier 0 networks and Tier 1plus networks routing in the first and second modes. In otherembodiments, content originators could also route in two modes whenthere is diversity in their interconnected delivery networks. Also,terminal networks could use the two mode routing when making contentobject requests of interconnected networks. Indeed, any networks thathave diversity in their interconnects could route in the two modes basedupon a utilization determination. In one embodiment, two mode routingallows avoiding bottlenecks.

While the principles of the disclosure have been described above inconnection with specific apparatuses and methods, it is to be clearlyunderstood that this description is made only by way of example and notas limitation on the scope of the invention.

What is claimed is:
 1. A network system that delivers content objectsfrom one or more content providers over the Internet to one or more endusers, comprising: a content request interface for receiving contentobject requests for delivery of content objects to the one or more endusers; and a traffic distributor configured to determine an efficientrouting path from one of the content providers to one of the end usersbased on efficiency and without considering effects of loading, byselecting among a plurality of full-route networks of the Internet,comprising a first full-route network and a second full-route network,to establish the efficient routing path, wherein the efficient routingpath includes at least one of the first and second full-route networks,an egress point of the first full-route network to a terminal networkthat serves the one of the end users, and an egress point of the secondfull-route network to the terminal network; process content objectrequests received from the one of the end users through the contentrequest interface such that the at least one of the first and secondfull-route networks is utilized; detect a routing bottleneck in theefficient routing path, wherein: the routing bottleneck is associatedwith a routing element that comprises at least one of the first andsecond full-route networks, the egress point of the first full-routenetwork, and the egress point of the second full-route network, and therouting bottleneck is detected based on utilization of the routingelement in excess of a threshold; determine a sub-optimal routing pathfrom the one of the content providers to the one of the end users thatis less efficient under light load conditions than the efficient routingpath, but does not include the routing bottleneck; process contentobject requests received from the one of the end users through thecontent request interface while utilizing the sub-optimal routing path;detect that the utilization of the at least one of the first and secondfull-route networks, the egress point of the first full-route network,and the egress point of the second full-route network has fallen belowthe threshold; and resume processing content object requests receivedfrom the one of the end users through the content request interfacealong the efficient routing path.
 2. The network system that deliverscontent objects from the one or more content providers over the Internetto one or more end users as recited in claim 1, wherein the efficientrouting path includes network hops, and the traffic distributordetermines the efficient routing path based on a number of the networkhops.
 3. The network system that delivers content objects from the oneor more content providers over the Internet to one or more end users asrecited in claim 1, wherein the efficient routing path includes networkhops, and determining the efficient routing path comprises determiningthe efficient routing path based on speed of one or more routers betweenthe network hops.
 4. The network system that delivers content objectsfrom the one or more content providers over the Internet to one or moreend users as recited in claim 1, wherein the efficient routing pathconnects one of the content providers directly with one full-routenetwork and connects the egress point of the one full-route networkdirectly with the terminal network.
 5. The network system that deliverscontent objects from the one or more content providers over the Internetto one or more end users as recited in claim 4, wherein the sub-optimalrouting path connects the one of the content providers with the onefull-route network, and connects the egress point of the one full-routenetwork with at least a second full-route network before connecting withthe terminal network.
 6. The network system that delivers contentobjects from the one or more content providers over the Internet to oneor more end users as recited in claim 1, wherein the sub-optimal routingpath replaces the routing element associated with the routing bottleneckwith a more costly routing element.
 7. The network system that deliverscontent objects from the one or more content providers over the Internetto one or more end users as recited in claim 1, wherein the sub-optimalrouting path is determined from a plurality of sub-optimal routing pathsthat differ by random assignment of differing full-route networks amongthe plurality of sub-optimal routing paths.
 8. The network system thatdelivers content objects from the one or more content providers over theInternet to one or more end users as recited in claim 1, wherein thetraffic distributor determines the sub-optimal routing path after acontent object request is received from the one of the end users throughthe content request interface.
 9. The network system that deliverscontent objects from the one or more content providers over the Internetto one or more end users as recited in claim 1, wherein determining thesub-optimal routing path includes determining a most efficient egresspoint for delivery to the one of the end users.
 10. A method ofoptimizing network delivery of content objects from one or more contentproviders over the Internet to one or more end users, comprising:determining, utilizing a traffic distributor, an efficient routing pathfrom one of the content providers to one of the end users based onefficiency and without considering effects of loading, wherein: thetraffic distributor selects among a plurality of full-route networks ofthe Internet, comprising a first full-route network and a secondfull-route network, to establish the efficient routing path, and theefficient routing path includes at least one of the first and secondfull-route networks, an egress point of the first full-route network toa terminal network that serves the one of the end users, and an egresspoint of the second full-route network to the terminal network;processing content object requests from the end users such that the atleast one of the first and second full-route networks is utilized;detecting, utilizing the traffic distributor, a routing bottleneck inthe efficient routing path, wherein: the routing bottleneck isassociated with a routing element that comprises at least one of thefirst and second full-route networks, the egress point of the firstfull-route network, and the egress point of the second full-routenetwork, and the routing bottleneck is detected based on utilization ofthe routing element in excess of a threshold; determining, utilizing thetraffic distributor, a sub-optimal routing path from the one of thecontent providers to the one of the end users that is less efficientunder light load conditions than the efficient routing path, but doesnot include the routing bottleneck; processing content object requestsfrom the one of the end users while utilizing the sub-optimal routingpath; detecting, utilizing the traffic distributor, that the utilizationof the at least one of the first and second full-route networks, theegress point of the first full-route network, and the egress point ofthe second full-route network has fallen below the threshold; andresuming processing content object requests from the end users along theefficient routing path.
 11. The method of optimizing network delivery ofcontent objects from the one or more content providers over the Internetto one or more end users as recited in claim 10, wherein the efficientrouting path includes network hops, and the traffic distributordetermines the efficient routing path based on a number of the networkhops.
 12. The method of optimizing network delivery of content objectsfrom the one or more content providers over the Internet to one or moreend users as recited in claim 10, wherein the efficient routing pathincludes network hops, and determining the efficient routing pathcomprises determining the efficient routing path based on speed of oneor more routers between the network hops.
 13. The method of optimizingnetwork delivery of content objects from the one or more contentproviders over the Internet to one or more end users as recited in claim10, wherein the efficient routing path connects one of the contentproviders directly with one full-route network and connects the egresspoint of the one full-route network directly with the terminal network.14. The method of optimizing network delivery of content objects fromthe one or more content providers over the Internet to one or more endusers as recited in claim 13, wherein the sub-optimal routing pathconnects the one of the content providers with the one full-routenetwork, and connects the egress point of the one full-route networkwith at least a second full-route network, before connecting with theterminal network.
 15. The method of optimizing network delivery ofcontent objects from the one or more content providers over the Internetto one or more end users as recited in claim 10, wherein the sub-optimalrouting path replaces the routing element associated with the routingbottleneck with a more costly routing element.
 16. The method ofoptimizing network delivery of content objects from the one or morecontent providers over the Internet to one or more end users as recitedin claim 10, wherein determining the sub-optimal routing path comprisesdetermining a plurality of sub-optimal routing paths that differ byrandom assignment of differing full-route networks among the pluralityof sub-optimal routing paths.
 17. The method of optimizing networkdelivery of content objects from the one or more content providers overthe Internet to one or more end users as recited in claim 10, whereindetermining the sub-optimal routing path occurs after a content objectrequest is received from the one of the end users.
 18. The method ofoptimizing network delivery of content objects from the one or morecontent providers over the Internet to one or more end users as recitedin claim 10, wherein determining the sub-optimal routing path includesdetermining a most efficient egress point for delivery to the one of theend users.
 19. A computer readable device with instructions foroptimizing network delivery of content objects from one or more contentproviders over the Internet to one or more end users, comprisingcomputer-executable code for: determining, utilizing a trafficdistributor, an efficient routing path from one of the content providersto one of the end users based on efficiency and without consideringeffects of loading, wherein the traffic distributor selects among aplurality of full-route networks of the Internet, comprising a firstfull-route network and a second full-route network, to establish theefficient routing path, and the efficient routing path includes at leastone of the first and second full-route networks, an egress point of thefirst full-route network to a terminal network that serves the one ofthe end users, and an egress point of the second full-route network tothe terminal network; processing content object requests from the endusers such that the at least one of the first and second full-routenetworks is utilized; detecting, utilizing the traffic distributor, arouting bottleneck in the efficient routing path, wherein: the routingbottleneck is associated with a routing element that comprises at leastone of the first and second full-route networks, the egress point of thefirst full-route network, and the egress point of the second full-routenetwork, and the routing bottleneck is detected based on utilization ofthe routing element in excess of a threshold; determining, utilizing thetraffic distributor, a sub-optimal routing path from the one of thecontent providers to the one of the end users that is less efficientunder light load conditions than the efficient routing path, but doesnot include the routing bottleneck; processing content object requestsfrom the one of the end users while utilizing the sub-optimal routingpath; detecting, utilizing the traffic distributor, that the utilizationof the at least one of the first and second full-route networks, theegress point of the first full-route network, and the egress point ofthe second full-route network has fallen below the threshold; andresuming processing content object requests from the end users along theefficient routing path.